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Abstract: This paper explores a new approach to the evaluation of educational technologies by 
using Eisner’s connoisseurship model. This model has two essential features in program 
evaluation: holistic approach to the analysis and interpretation of data and multiple perspectives 
in the evaluative tasks (Eisner, 1998). Using Eisner’s model, the key technology connoisseurs at 
a state university in the Southeast were interviewed to shed light on the evaluation practices of 
educational technologies. The results of this study are summarized for those who are interested 
in alternative evaluation models. The findings should also serve as a guide for technology leaders 
in making educational improvements. 



Summary 



Purpose 

Local and state agencies as well as the federal government have invested heavily in 
various technology tools for schools and classrooms for the last two decades. Meanwhile, efforts 
that demonstrate the value of educational technologies in the learning environment gained 
momentum by increasing the discussions on technology’s effectiveness in schools (Haertel and 
Means, 2003). In this regard, the call for ‘evidence’ for effectiveness became a buzzword and 
researchers started conducting studies to better guide the technology investments. 

Ling (n.d., ( ][2) believes that there are four positions in the evaluation of educational 
technologies: One is oriented to the stated objectives of the particular educational innovation; 
one to comparison with alternative educational approaches; one to the benefits and costs 
anticipated from a knowledge of the state of the art of learning technologies; and one to criteria 
developed from a particular educational and/or social theory. This paper will use the fourth 
position while evaluating effectiveness of technologies in one university’s campus. 

Theoretical Framework: Connoisseurship Model 

Eisner’s connoisseurship model provides a strong framework and alternative to the 
experimental design evaluations. According to Eisner (1998), connoisseurship is “the art of 
appreciation” (p.63), and educational connoisseurs are those who have a distinctive awareness of 
qualities in different settings. He thinks connoisseurs evaluate the issues by looking at their 
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qualities and illuminate a situation so that it can be seen or appreciated by others as well. In the 
evaluation of educational technologies, the connoisseurship model has two major implications: 
holistic approach to the analysis and interpretation of data and multiple perspectives in the 
evaluative tasks. Holistic approach guides researchers from the formation of data collection 
instruments to the final write-up. 

Eisner points out five dimensions of schooling that educational connoisseurs consider: 
intentional, structural, curricular, pedagogical and evaluative. The “intentional level deals with 
goals or aims that are formulated for the school or a classroom” (Eisner, 1998, p.73). The 
structural level addresses the organizational forms of the schools and classrooms such as 
facilities, time, utilities and their cost. The curricular dimension focuses upon “the quality of the 
curriculum’s content, goals and activities employed to engage students in it” (Eisner, 1998, 
p.75). The pedagogical level deals with teaching and learning occurred in the classrooms. The 
last dimension, the evaluative level emphasizes the evaluation practices that influence the 
learning environment. 



Data Collection Process 

The data in this study were collected through open-ended interviews with 7 technology 
connoisseurs in a public university in the Southeast. In the selection of participants, an email that 
explains the purpose of the study was sent to all technology faculty and staff in the university. 
Only those who were volunteered and has available time for an interview joined to this study. 
Participants range from technology support specialists for different colleges in the campus to 
information technology service directors, from university faculty to technology trainers who are 
well known at this institution and whose responsibility is to use and integrate educational 
technologies. 

The interview questions were developed by the researchers using five dimensions of the 
connoisseurship model. Each interview took 25-30 minutes and was audio-taped with the 
consent of the participants. Interviews took place in fall, 2004. The recorded interviews are 
transcribed for further data analysis. After this task was completed, researchers used the holistic 
approach to qualitative data analysis, and summarized the results. 

Results 

The first section of the interview contained warm-up questions of which probed 
participants’ demographic information. Table 1 is to summarize these results. 
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Table 1: Demographics 



Position 


• Tenured faculty member = 3 

• Technology Administrator = 4 


Years at the Institution 


• 1-5 years = 2 participants 

• 6-10 years = 0 participants 

• 11-15 years = 3 participants 

• 16-20 years = 1 participant 

• 21-25 years = 1 participant 


Gender 


• Female = 3 

• Male = 4 



The second section of the interview schedule included two questions. The first one was 
about the criteria used in the selection of educational technologies, and second one was about 
whether technology investments or available technology in classrooms match the technology 
goals of the university. Almost all participants concluded that use of technology in the 
classrooms to support instruction is number one criteria to make decisions in technology 
investments. One interviewee mentioned that if a technology is deemed important by the faculty 
members, even if is not vitally used, they usually have very good support from the 
administration. This shows that faculty members at the higher education institutions are very 
critical to set up the criteria for technology. Another participant mentioned that use of standard 
software and hardware across the campus is very important criteria while purchasing new 
software. He also added that at the moment because of this criteria faculty members, staff and 
students can walk from one unit to another and use the same program. 

In terms of second question in this section, one participant mentioned university’s butter 
and bread goals and the part about technology: 

To achieve national recognition, the University should (a) develop high-tech minors and 
programs for students in the humanities, natural and social sciences, business, and 
education; (b) ensure that every graduate is competent in the use of information 
technology in his or her discipline, and (c) become a leader in the innovative use of 
asynchronous learning environments to support both distance education and classroom 
instruction. 

Almost all participants agreed that use of technology in the broad sense of the word 
matches with the university’s mission. Table 2 summarizes the rest of the results for this section. 
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Table 2: Intentional Level: Goals, aims and intentions 



Criteria used to choose software and hardware 

• Technology’s use in the schools; 

• Future use of technology in the schools; 

• Cost of technology; 

• Compatibility of technology hardware and software with the technology that is 
currently in place; 

• The technology plan and what is proposed to be used and purchased in the future due 
to changes taking place; 

• Easy support; 

• Our criteria is for whole campus software and hardware; peer institutions may serve as 
models; 

• The extent technology will support instruction; 

• Broad use of technology; 

• Faculty members’ requests based on curricular needs; 

• Criteria are not clear all the time; 

• Outside agencies that rank the universities set some of the criteria. 

• Higher Education IT groups such as Educause 

• University’s bread and butter goals. 

• Budget. 

• The University Technology Committee. 

• Extent of technology use. 

• The consistency of technology use; 

• The portability of the technology; 

• The use of standard software programs across the campus; 

• The software sets the criteria for hardware; 

• The criteria set up by lower level technology groups across the campus 

• Software and hardware should meet pedagogical needs; cost; 

• The skills of the people who are going to use technology 
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Match between technology and university goals 

Interviewee #1: There is a close match: Department focuses on students’ effective use of 
technology and becoming a leader in the school; effective and innovative use of 
asynchronous learning environments that support classroom instruction and distance 

education. 

Interviewee #2: We follow university’s bread and butter goals for technology; individual 

units match their goals with the university’ goals. 

Interviewee #3: Everything we do is to advance university’ mission; university’ goals are 

consideration for us before we implement anything. 

Interviewee #4: We try to match up enterprise applications for entire university. 

Interviewee #5: This is a technology rich institution. Availability helps reaching the 
university’s goals. 

Interviewee #6: University goals are very broad and open when it comes to technology. 
University Technology Planning Committee has a little more specific IT Strategic Plan. It 

is easy to fit in there. 

Interviewee #7: My observation is that people have bought the technology and then tried to 
find a need for it. 



When interviewees asked for the available technology that facilitates successful 
technology integration, they all agreed that university has enough resources. One interviewee 
stated that their university has a number of computer labs available on campus for use by 
students and faculty for computing classes and individual use to work on technology projects as 
well as a wealth of technology rich classrooms and resource rooms such as the Teaching 
Materials Center, Learning Resource Center, Video Editing rooms, Darkrooms, Educational 
Technology Training Center, and multimedia rooms. In terms of offering technology training, 
same participant stated that a wealth of technology and distance training classes were offered to 
faculty across the colleges to enhance faculty skills and knowledge. Many of these are 
collaborative training sessions across colleges and departments. Another participant stated that 
the university has adapted a more decentralized model of technology support in the past few 
years but different IT units collaborate, share information and work together for a common 
cause. Table 3 summarizes results of this section. 
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Table 3: Structural Level: University’s Organizational Structure 



In what ways does the organizational structure of the university (classrooms, labs, 
utilities, time allocation for courses, subject areas technology used, degree levels) 
facilitate successful technology integration? 



• A number of computer labs are available for students; 

• Technology rich classrooms are available for teachers and students to use; 

• There are a wealth of technology training opportunities offered to both students and 
faculty; 

• The organization provides many online resources for students and faculty; 

• The Distance Education Office provides incentives to distance faculty; 

• There are special courses for technology integration; 

• A student computing club exists for interested students; 

• Electronic portfolios that show students’ technology skills; 

• Each college has their own resources, budget, and planning committees; 

• One on one training provided for the faculty; 

• Assistance with technology is provided; 

• Decentralized model of technology support is helpful; 

• This university has done a pretty good job providing the resources. 



The fourth section of the interview is about curriculum content, goals, and activities. 
When participants asked for evidence whether technology matches with the curricular goals or 
not, one participant stated that this process is very informal. She said: 

If things are going along smoothly and you don’t hear many complain then you know that 
things are matching pretty well. If you hear rumblings and grumbling about the 
technology, then you know that there are problems. 
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Table 4 summarizes how well participants felt technology matched curricular content and 
evidence that supported this and successful technology integration. 

Table 4: Curricular Level: Curriculum Content, Goals and Activities 



Match with technology and curricular content 

• # of participants who reported there was a match = 4 

• # of participants who did not = 2 

• # of participants do reported there was not match = 1 

Evidence demonstrating technology matches curricular content 

• Formal and informal feedback from students; 

• Formative and summative course evaluations; 

• Informal dialogue between department members; 

• Professors inform technology leaders; 

• Student’s work and electronic portfolios; 

• Students complain when technology does not match the curriculum content; 

Evidence technology is successfully integrated into the curriculum 

• Class observations by the administrator; 

• Listening to student questions and concerns; 

• Projects created by students; 

• Faculty web sites; 

• Informal dialogues with faculty members; 

• Informal dialogues with students; 

• Annual reports from each department and college; 

• Observations of students teaching; 



The fifth section of the study targeted learning and teaching experiences. Most of the 
participants stated that individual faculty members and departments set up the criteria for 
evaluation and there is no standardization across the campus in this regard. Other results of this 
section are summarized in Table 5. 
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Table 5: Pedagogical Level: Learning and Teaching 



Criteria used to determine effective technology in enhancing learning 

• Research that shows positive impact of technology on student learning; 

• There are a number of people impacted by certain technology; 

• The cost of technology; 

• Best practices using technology; 

• Replacement of old technologies; 

• Multimedia features of computers enhance learning; 

• Observing students and their use of technology in the field after graduation; 

Criteria used to determine effective technology in enhancing teaching 

• Technology’s capability to enhance student’s ability to learn concepts; 

• Technology’s capability in meeting diverse and varying needs of the students; 

• Number of people impacted by certain technology; 

• Cost; 

• Replacement of old technologies; 

• Best practices using technologies; 

• University Technology Committee discussions provide leadership; 

• Published research studies on criteria to use to determine effective technology in 
teaching; 

• Student feedback; 

• Real life needs in the teaching situation; 

Role of instructors in effectively integrating technology 

• Model effective technology use; 

• Using new and emerging technology; 

• Faculty collaborating with instructional designers in their teaching; 

• Faculty serving as a facilitator of student learning; 

• Faculty should have good technology skills and understand what effective technology 
integration is; 



The last section of the interview dealt with the evaluation practices for effectiveness of 
educational technologies. Results of this section can be viewed in Table 6. 
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Table 6: Evaluative Level: Evaluation Practices 



Questions 


Evaluation practices of the 
educational technologies 


Evaluation practices that 
take place in the university 


Interviewee 1 


• University course evaluations; 

• University distance education course 
evaluation; class observations; 

• Formative and summative course 
evaluations; 

• Department programs outcomes 
survey; 

• Program completers survey; 

• Mid program survey; 

• Feedback gotten at the oral exams. 


• Focus group interviews 
with the students; 

• Focus group interviews 
with the cohorts; 

• Needs assessment survey 
administered by some 
faculty; 

• COE Technology 
Committee reports; 

• University Technology 
Committee reports; 

• Learning Resource Center 
surveys; 

• Distance Education Office 
surveys. 


Interviewee 2 


• University Technology Planning 
Committee’s annual reports; 

• Instructional Technology Services 
(ITS) Annual report (not directly 
related to educational technologies 
but it is evaluation of overall 
technology in the campus) 


• University Technology 
Planning Committee, 

• Strategic Plan 


Interviewee 3 


• Formalized survey of the students 
(sometimes students don’t take this 
seriously, so informal feedback 
should be considered as well); 

• Formalized survey of the teachers; 

• Informal email sent to the faculty 
members. 


• Formation of different 
groups that meet on 
regular basis to discuss 
technology on campus (in 
this university not 
everyone agree on how we 
do things but we do all 
communicate about it) 


Interviewee 4 


• We track use of certain technologies 
across the campus 


• University Strategic Plan 
for technology. 


Interviewee 5 


• We have to do more evaluation 
research. Unfortunately most 
faculties use technology and believe 
that that is effective and fine and they 
don’t need to do anything else. 


• I see a genuine concern to 
make technology available 
everywhere. Accreditation 
reports show this. But 
there is no systematic 
formal evaluation of the 
technologies campus-wide. 
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Interviewee 6 


• Formal surveys; 

• Personal observations. 


• Feedback through small 
studies; 

• Quick polls; 

• Decentralized units. 


Interviewee 7 


• Student surveys and interviews; 

• Interviews of alumni; 

• External interviews. 


• Very mixed. Some 

departments do abetter job 
than the others. 



Importance of the Study 

We believe this study could serve educational leaders in higher education a multitude of 
ways. The information revealed from the seven participants may cause technology administrators 
and/or teachers to take a closer look at their technology systems, especially their teaching, 
learning, and evaluation practices in order to make program improvements. The preliminary 
findings may provide some additional ways of examining our current practices. 
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